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EXECUTIVE  SUMMARY 


The  Federal  Aviation  Administration  (FAA)  is  evaluating  the  use  of  digitized 
voice  for  air-to-ground  air  traffic  control  (ATC)  traffic.  This  technique  offers 
the  promise  of  being  able  to  reduce  some  of  the  adverse  affects  of  voice 
transmission  media  on  the  quality  of  the  voice  traffic.  By  digitizing  the  voice, 
the  apparent  quality  of  the  voice  signals  will  be  more  consistent  for  the  ATC 
controller . 

In  support  of  this  effort,  preliminary  tests  were  conducted  in  August  1988  on  a 
number  of  commercially  available  digital  voice  coder/decoders  (CODECS) .  This 
report  contains  preliminary  results  of  the  tests  which  were  conducted  on  CODECS 
from  a  number  of  manufacturers  using  air  traffic  controller  personnel  from  the 
New  York  ATC  facility  in  Long  Island  as  listeners. 

The  tests  were  based  on  a  series  of  standardized  messages  digitized  by  the 
various  CODECS.  The  subsequently  reconstructed  voice  messages  were  recorded  and 
replayed  for  the  controllers  who  made  subjective  assessments  of  the  qualify  of 
each  message.  The  test  results  were  evaluated  with  the  objective  of  identifying 
the  better  performing  (CODEC)  of  the  test  lot  for  further  study. 

The  best  performing  CODEC  was  the  9.6  kilo  bit  per  second  (kbps).  The  best  4.8 
kbps  CODEC  was  ranked  slightly  lower  than  the  9.6  kbps  CODEC  indicating  potential 
use  of  4.8  kbps  for  ATC  applications. 
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1.  TEST  PROGRAM  OBJECTIVES. 


The  principal  objective  of  the  tests  was  to  evaluate  the  performance  of  various 
coder/decoders  (CODECS)  for  use  in  an  air  traffic  control  (ATC)  environment  using 
air  traffic  controllers, 

1.1  BACKGROUND. 

Voice  CODECS  are  being  considered  for  use  in  air-to-ground  communications  via 
satellite.  As  there  are  a  large  number  of  CODEC  manufacturers  in  the  market, 
there  is  a  need  to  identify  the  highest  performing  low  data  rate  models  for 
further  consideration.  The  lower  data  rate  is  sought  due  to  the  resulting 
improvements  in  the  radio  frequency  (RF)  link  power  budget.  This,  in  turn,  can 
ultimately  affect  the  cost  of  implemented  systems.  During  the  tests  the  CODECS 
were  evaluated  for  intelligibility  as  well  as  acceptability  on  the  part  of  ATC 
personnel . 

Although  low  data  rate  CODECS  may  be  used  with  some  degradation  in  the  telephony 
environment,  their  use  in  the  ATC  environment  must  not  be  allowed  to  jeopardize 
air  safety.  In  order  to  address  the  numerous  subjective  performance  issues 
associated  with  the  use  of  voice  CODECS,  air  traffic  controllers  were  used  to 
conduct  subjective  listening  tests.  The  results  of  this  initial  test  program 
are  intended  to  lead  to  more  rigorous  evaluations  of  a  smaller  number  of  CODEC 
models  in  the  near  future.  Such  tests  will  study  the  performance  under  high 
noise  environments  and  address  a  number  of  technical  issues  such  as  the  voice 
power  spectrum  of  reconstructed  voice  signals. 

2.  TEST  METHODOLOGY. 

Testing  conducted  in  this  initial  phase  of  the  CODEC  evaluations  placed  minimum 
operational  stress  on  the  CODECS  and  did  not  involve  noisy  environments  or 
noticeably  accented  voices,  both  of  which  can  place  additional  constraints  on  the 
usefulness  of  voice  digitizing,  low  data  rate  CODECS.  Therefore,  the  tests 
helped  only  to  identify  the  CODECS  that  have  acceptable  voice  processing 
algorithms  under  ideal  operating  conditions. 

ATC  controllers  were  used  for  the  tests  as  opposed  to  professional  "listeners," 
who  are  typically  oriented  to  telephony  performance  evaluation.  ATC  voice 
traffic  characteristics  vary  from  those  of  normal  telephony.  The 
intelligibility  requirements  are  very  high  in  the  ATC  environment.  Further, 
traffic  is  largely  "in  context"  or  anticipated  by  the  controllers  which  can  bias 
the  intelligibility  of  a  message.  Thus,  it  is  necessary  to  evaluate  the  CODECS 
with  a  focus  on  ATC  services. 

Controllers  are  well  trained  in  working  over  circuits  with  considerable 
degradation.  The  types  of  signal  impairments  experienced  by  many  of  them  to  date 
differ,  however,  from  the  types  of  impairments  experienced  through  digital  voice 
CODECS.  There  is  some  question  concerning  the  subjective  sensitivity  of 
controllers  to  CODEC  performance  with  and  without  noise  impairments.  This 
suggests  that  controllers  be  used  for  future  listening  tests. 
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2.1  TEST  PROCEDURES. 

Two  types  of  messages  were  used  for  the  tests.  One  type  was  taken  from  a  set  of 
standard  phonetically  balanced  voice  messages  known  as  Dynamic  Acceptability 
Measurement  (DAM) .  A  second  type  of  message  consisted  of  recordings  of  actual 
ATC  voice  traffic  recorded  at  the  Leesburg  ATC  facility.  The  test  material  and 
evaluation  foirms  are  presented  in  the  appendix.  The  use  of  ATC  voice  traffic  is 
considered  more  pertinent  to  the  present  evaluation  of  CODECS.  Also,  the  use  of 
the  two  types  of  messages  will  help  to  relate  the  results  of  independent  tests  by 
others  for  telephony  applications  to  the  ATC  environment. 

The  test  voice  messages  used  for  the  tests  were  recorded  on  a  high  quality  audio 
cassette.  Copies  of  the  cassette  were  submitted  to  a  number  of  CODEC 
manufacturers,  who,  in  turn,  processed  the  test  messages  through  their  CODEC. 

The  processed  voice  messages  were  then  assembled  in  a  test  sequence  for  replaying 
to  the  test  controllers.  The  test  messages  are  grouped  into  16  sets  of  several 
sentences  each.  Each  set  had  previously  been  processed  through  14  CODECS.  The 
entire  set  of  resulting  test  messages  consists  of  240  digitized/reconstructed 
messages  plus  the  set  of  16  original,  unprocessed  voice  messages.  These  256  sets 
were  then  randomly  rearranged  and  played  to  the  listening  audience  of  ATC 
controllers . 

The  principal  performance  issue  of  CODECS  as  related  to  use  by  ATC  personnel  is 
intelligibility.  Also  of  importance  is  acceptability  on  the  part  of  the 
controllers.  Voice  that  is  digitized  into  low  data  rates  signals,  i.e.,  4800 
bits  per  second  (bps) ,  has  a  "signature"  or  audio  character  that  may  or  may  not 
be  burdensome  to  the  controller.  The  test  controllers  were,  therefore,  asked  to 
score  both  the  intelligibility  and  acceptability  of  each  test  message.  Although 
not  rigorous,  the  scoring  system  does  provide  results  suitable  to  rank  the 
various  CODECS  in  order  of  utility  in  the  ATC  environment. 

In  the  tests,  CODEC  models  G  and  L  were  inserted  to  provide  control  messages. 
Model  G  is  the  original,  unprocessed  voice.  Model  L  is  a  9.6  kilobit  (kb)  CODEC 
with  near  "toll  quality  telephone"  performance  characteristics. 

The  tests  were  conducted  at  the  New  York  ATC  facility  during  early  August  1988. 
Appendix  A  contains  a  transcript  of  the  voice  test  messages,  test  procedure 
details,  and  a  sample  of  the  scoring  sheets  used  to  record  test  result.  The 
testing  took  about  1  hour  and  20  minutes  plus  break  periods  approximately  every 
half  hour. 

3.  EVALUATION  OF  TEST  RESULTS. 

3.1  SUMMARY  OF  TEST  RESULTS. 

The  total  number  of  discrete  scores  recorded  are  3840  for  intelligibility  and 
3840  for  acceptability.  That  is,  16  listeners  scored  16  messages  through  15 
CODECS.  These  are  summarized  in  figure  1.  The  figure  shows  the  average 
intelligibility  and  acceptability  scores  for  all  CODECS.  CODEC  F  had  the  highest 
scores  of  the  4.8  and  2.4  kb  CODECS. 
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3.2  ACCEPTABILITY  SCORES  VS.  INTELLIGIBILITY  SCORES. 

Figure  2  shows  the  intelligibility  and  acceptability  scores  for  all  CODECS  in 
order  of  scores.  Both  types  of  scores  tend  to  rank  the  CODECS  in  the  same  order. 
The  correlation  of  the  intelligibility  and  acceptability  scores  can  be  seen  in 
figure  3.  It  can  be  concluded  that  the  listeners  were  in  close  agreement  between 
intelligibility  and  acceptability  scores. 

3 ■ 3  DATA  INTEGRITY . 

The  validity  of  a  set  of  subjective  test  data  can  often  be  validated  by 
assessing  the  distribution  of  the  data.  The  distribution  of  intelligibility 
scores  is  shown  in  figure  4.  This  figure  shows  the  standard  deviation  of 
listener  scores  for  each  message-CODEC  test.  There  are  240  points  in  the  graph 
representing  15  CODECS  times  16  message  sets.  The  data  points  appear  to  be 
approximately  a  (Gaussian)  distribution  around  a  mean  of  0.7  score  points  (0.7 
out  of  a  range  of  5.0  suggests  reasonable  integrity  for  this  type  of  testing). 

Figures  5,  6,  and  7  show  the  average  deviation  for  each  listener  from  the  group's 
average  score  during  the  testing.  Listeners  1  and  5  seem  to  have  changed  their 
general  attitude  during  the  test  from  one  extreme  to  the  other.  Listener  11 
remained  optimistic  throughout  the  period.  In  paragraph  3.4  below  further 
consideration  is  given  to  these  anomalies. 

Further  confirming  the  validity  of  the  test  results  are  the  standard  deviation 
of  intelligibility  scores  shown  in  figure  8  by  CODEC  model.  The  deviation  per 
quarter  of  the  testing  period  for  each  CODEC  is  a  fairly  narrow  range.  It  is 
noted  that  the  deviation  of  opinions  is  lowest  for  the  clear,  unprocessed  voice. 

3.4  EVALUATION  BY  MESSAGE  TYPE. 

Figure  9  shows  the  intelligibility  scores  as  a  function  of  message  type.  In 
general,  female  voices  were  rated  lower.  Likewise,  ATC  messages  received  lower 
scores  than  the  standard  DAM  messages.  This  is  a  non- trivial  point,  since 
evaluation  by  DAM  messages  alone  will  probably  not  produce  test  results 
indicative  of  the  performance  in  an  ATC  environment.  Figure  10  further  confirms 
this  by  showing  that  unprocessed  DAM  messages  score  higher  than  unprocessed  ATC 
messages.  The  higher  score  of  DAM  messages  for  clear  voice  indicates  that  ATC 
messages  may  be  inherently  more  difficult  to  understand. 

Figures  11  and  12  show  that  the  listener  group  had  a  high  degree  of  agreement  on 
the  scoring  of  message  13.  Message  13  is  a  female  speaker  giving  an  ATC 
message.  This  further  substantiates  the  importance  of  using  ATC  voice  traffic 
for  the  CODEC  tests.  Figure  13  compares  ATC  and  DAM  messages  through  the 
various  CODECS.  In  some  cases  there  is  a  significant  difference  between  the  ATC 
and  DAM  messages.  See  CODEC  "N"  where  message  13  scores  1.3  versus  2.8  for  DAM 
messages . 

3.5  EVALUATION  BY  CODEC/MESSAGE  TYPE. 

Figure  14  shows  the  intelligibility  scoring  for  each  CODEC  versus  speaker  gender 
and  message  type  during  each  of  the  four  quarters  of  the  test  period.  This 
demonstrates  that  each  manufacturers  algorithm  will  handle  voice  differently 
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depending  on  the  characteristics  of  the  message.  These  test  evaluations  will  be 
used  to  help  structure  subsequent  CODEC  tests. 

3.6  EVALUATION  BY  TEST  LISTENER. 

Figure  15  shows  the  variation  of  intelligibility  scores  as  a  function  of 
listener  years  of  experience.  A  similar  curve  of  acceptability  scores  (not 
included)  shows  essentially  the  same  results.  The  figure  shows  a  slight 
tendency  for  personnel  in  the  middle  of  the  range  to  have  higher  intelligibili  ty 
scores.  Figure  16  is  based  on  the  same  data  used  to  generate  figure  15,  but 
separates  the  data  by  gender  of  the  speaker.  From  the  figure  it  becomes  evident 
that  the  gender  of  the  speaker  does  not  significantly  affect  intelligibility 
versus  age  of  the  listener. 

The  listeners  varied  in  average  score  as  seen  in  figure  17.  For  example, 
listener  11  (49  years  of  age)  had  a  significantly  higher  average  score  for  both 
intelligibility  and  acceptability  messages. 

A  question  of  data  integrity  is  suggested  by  the  variations  in  performance  by 
listeners.  Figures  18  and  19  show  the  scores  for  intelligibility  and 
acceptability  using  a  subset  of  the  listeners.  To  establish  this  subset,  the 
listeners  in  the  middle  age  group  were  selected.  As  much  as  half  a  point 
difference  can  be  seen  between  this  group  and  the  overall  group.  This  suggests 
that  toleration  of  CODECS  by  ATC  personnel  may  vary  significantly. 

4.  CONCLUSIONS. 

Coder/decoder  (CODEC)  models  F  (4.8  kilo  bit  per  second  (kbps)),  A  (4.8  kbps), 

B  (4.8  kbps),  and  D  (4.8  kbps)  are  the  highest  ranked  CODECS  as  being  most 
acceptable  to  air  traffic  controllers.  These  can  be  further  studied  in  depth  for 
performance  under  more  stressful  operating  conditions,  for  example,  various  bit 
error  rates,  high  background  multiple  speaker,  and  non- speech  sound  levels. 

5 ■  RECOMMENDATIONS . 

The  principal,  industry-wide  issue  of  performance  among  digitizing  coder/ 
decoders  (CODECS)  is  immunity  to  non-voice  interference.  Audio  background  noise 
can  seriou.sly  affect  the  performance  of  a  CODEC.  If  a  processing  algorithm  does 
not  properly  account  for  impulse  as  well  as  steady  state  background  noise,  voice 
intelligibility  can  be  easily  degraded.  Further  studies  should  evaluate  CODECS 
F,  A,  B,  and  D  under  various  noise  conditions.  Robustness  of  these  CODECS  may  or 
may  not  be  well  correlated  to  overall  scores  achieved  in  this  initial  testing 
phase.  CODEC  models  H,  K,  N,  and  I  can  be  held  in  reserve  should  the  initial  set 
prove  to  be  unsuitable  for  air  traffic  control  (ATC)  use. 

A  CODEC's  processing  algorithm  should  also  consider  digital  transmission  error 
rates.  Error  correcting  functions  can  add  several  decibels  (dB)  of  dynamic  range 
to  a  communications  link.  Tests  should  be  conducted  with  bit  error  rates  on  the 
communications  circuit  of  up  to  10'^.  This  should  be  accomplished  using  the 
CODEC  test  bed  facility  during  the  next  phase  of  testing. 

The  issues  of  noise  as  well  as  voice  pattern  degradation  should  be  analyzed  for 
any  CODEC  implemented  in  the  ATC  environment.  The  dominant  criteria  for 
acceptance,  however,  should  remain  the  subjective  measure  of  intelligibility  by 
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ATC  personnel, 
the  next  set  of 
interference  to 


More  comprehensive  testing  methodology  should  be  developed  for 
CODEC  tests  which  adds  noise  and  multi -talker  background 
•i.jre  realistically  simulate  the  real  environment. 
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FIGURE  1  .  ACCEPTABILITY  AND  INTELLIGIBILITY  (AVERAGE) 
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FIGURE  2.  ACCEPTABILITY  AND  INTELLIGIBILITY  (INDIVIDUAL) 
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FIGURE  3.  COMPARISON  OF  INTELLIGIBILITY  TO  ACCEPTABILITY 
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FIGURE  4.  STANDARD  DEVIATION  OF  INTELLIGIBILITY  SCORES 
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STD  DE\/  lEJ  SCORE  ROlEJTS 


G,CLR  1,9.6  A,4.3  B,4.8  D,4.3  F,4.8  H,4.8  J,4,8  0,4.8  C,2.4  E,2.4  1,2,4  K,2.4 11,2.4  N,2.4 


S3  QIR 


COOK  m  BY  REFERENCE  NUMBER,  SPEED 
^  QTR2  ^0TR3 


E3 


FIGURE  8.'  STANDARD  DEVIATION  OF  INTELLIGIBILITY  SCORES  VS.  TEST  PERIOD 
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FEMALE  ATC 


MALEATC 


FEMALE  DAM 


MALE  DAM 


MESSAGE  lYPE 
Zh  AVERAGE  ALL  TESTS 


FIGURE  9.  INTELLIGIBILITY  SCORES  (AVERAGE  ALL  TESTS) 


MESSAGE  WE 
21  UNPROCESSED  VOICE 

10.  INTELLIGIBILITY  SCORES  (UNPROCESSED) 


SCORE  ROIEITS 
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-N 
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i 


ft 
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A 


N 
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\ 

V 
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A 


S 
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m 


1' 


Am 


AH 
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A 

A 

A 

A 

A 
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A 

A 

N 

t 

hA^ 

A 

A 
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A 

m 

A 

A 

hA 


AH 
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^A^ 

A 

N 

t 

hA^ 

A 

A 

A 


mA 

A 

A 

hA 

A 

A 

A 


A 

Na«| 

A 

/S 

k 


G.  CIEAEO.BF, 4.8 A, 4.8 8,4.80, 4.8H,4.8K,2.4N, 2,4 1,2.4  0,2.40,4.8 J, 4.3 E.2.4\<, 2.4 


IXU  Aa  TEST  OA  lA 
[5!  ATC  MESSAGES  ONLY 


CODEC  BY  REFERENCE  LETTER  AND  SPEED 
23  message  13  ONLY  ’  ' 


3  0AM  MESSAGES  ONLY 


FIGURE  13.  INTELLIGIBILITY  SCORES  VS.  MESSAGE  TYPE 
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IVI  UNPROCESSED  VOICE  ^  L,  9.6  ra  A. « 

E]  8. 4,8  ®  0, 4.8  ®  F.  4.8 

FIGURE  14.  INTELLIGIBILITY  SCORES  FOR  VARIOUS  MESSAGE  TYPES  (SHEET  1  OF  3) 
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MOIMIcdO  r'JV'31'M 


1  I  ^  I  \  I  1  i  I  I  \  I  ^  i  r 

1  2  3  4  5  6  7  8  9  10  n  12  13  14  15 

USTENER  BY  REFERENCE  NUMBER 


FIGURE  17.  AVERAGE  BIAS  IN  INTELLIGIBILITY  SCORES 


24 


CODEX  lYPE  BY  REFERENCE  NUMBER,  SPEED 


FIGURE  18.  AVERAGE  OF  DAM  INTELLIGIBILITY  SCORES 


APPENDIX  A 

SUBJECTIVE  LISTENING  TEST 


F, 


AA  CODED  SUBJECTIVE  USTEKllNS  EVALUATaOU 

for 

SATELLITE  LOW  HATE  VOICE  DEM0NSTHAT30K1  Am 
EVALyATiOKl  PHOCHAVl 


PROCEDURE 

You  will  be  asked  to  listen  to  number  of  digitally  processed  audio 
samples  through  a  set  of  headphones.  This  evaluation  will  consist  of  two 
parts.  Each  part  will  be  divided  up  into  fifteen  (15)  groups  (part  1  will 
consist  of  groups  1-15  and  part  2  of  groups  16  -  30).  Each  group  will 
consist  of  eight  (8)  samples  each.  Finally,  each  sample  will  contain  either  a 
set  of  three  sentences,  or  Air  Traffic  Control  (ATC)  dialogue.  Each  group 
will  be  arranged  according  to  the  following  sample  sequence; 


Sample  # 

Speaker  _ 

Material 

Duration 

1 

FEMALE  1 

3  SENTENCES 

10  SEC 

2 

MALE  1 

3  SENTENCES 

10  SEC 

3 

FEMALE  2 

3  SENTENCES 

10  SEC 

4 

MALE  ATC 

DIALOGUE 

20  SEC 

5 

FEMALE  ATC 

DIALOGUE 

20  SEC 

6 

MALE  2 

3  SENTENCES 

10  SEC 

7 

FEMALE  3 

3  SENTENCES 

10  SEC 

8 

MALE  3 

3  SENTENCES 

TOTAL 

10  SEC 
100  SEC 

Each  of  the  two  parts  will  last  approximately  40  minutes.  There  will  be 
a  15  minute  intermission  after  part  1. 

Please  listen  to  each  complete  sample  of  digitally  processed  audio, 
then  indicate  your  opinion  of  the  intellioibilitv  and  overall  acceptability  of 
sound  on  the  evaluation  forms  provided  to  you.  There  will  be  a  period  of 
silence  after  each  sample  for  your  opinion  score.  These  opinion  scores 
should  be  evaluated  based  on  the  following  five-point  scale: 

5  -  EXCELLENT 

4  -  GOOD 

3  -  FAIR 

2  -  POOR 

1  -  BAD 

Thank  you  for  your  cooperation! 


A-l 


SPEAKER 


MATERIAL 


FEMALE  (VW) 

P/0  DAM  List  6A2 

1) These  shoes  were  black  and  brown. 

2) They  are  too  loud  in  church. 

3) The  rabbits  and  dogs  drowned. 

MALE  (CH) 

P/0  DAM  List  4A 

1) The  girl  lost  the  foot  race. 

2) Card  games  are  fun  to  play. 

3) Happy  hour  is  over. 

FEMALE  (KS) 

P/0  DAM  List  8A2 

1) My  razor  gives  close  shaves. 

2) The  convicts  had  no  hope. 

3) l  jumped  on  the  new  bed. 

MALE 

ATC  Dialogue 

1) Eastern  731,  Atlanta  126.77  Good  Morning 

2) East6rn's  flight  101  Atlanta  Center  126.77 
Good  Morning 

3) 71 1  Mike  Alpha  maintain  flight  levels  240. 

4) US  Air's  135  Roger  contact  Washington 
Center  125.75. 

FEMALE 

ATC  Dialogue 

1)  Delta  799  heavy,  contact  Atlanta  Center 
126.77. 

2) LIS  Air  447  plus  30  miles  northeast  of 
Greensboro  at  one  1000  on  the  Greensboro 
altimeter,  3025. 

3) United  1199  turn  10  degrees  left,  intercept 
J48  on  the  Southwest  side  of  Montebello. 

4) Roger,  United  1199  climb  and  maintain 
level  350. 

MALE  (RH) 

P/0  DAM  List  6A 

1  )He  sprayed  our  house  for  bugs. 

2)  We  saw  a  bad  movie. 

3) That  hose  can  wash  her  feet. 

FEMALE  (MP) 

P/0  DAM  List  8A 

1 )  Don't  throw  trash  on  the  street. 

2) They  want  two  red  apples. 

3) Their  cooking  was  not  great. 

MALE  (JE) 

P/O  DAM  List  3A 

1  )Those  boxes  were  not  full. 

2) AII  the  boys  have  cold  feet. 

3) That  frog  jumped  through  the  weeds. 

ECT 


TEST 


R 


SAMPLE 

# 


1 


2 


3 


4 


5 


SPEAKER 


MATERIAL  FOR  PART  1  (G 


PUPS 


16-  2D\ 


MATERIAL 


FEMALE  (VW) 

P/0  DAM  List  6A2 

1) 1  suggest  you  leave  now. 

2) Music  can  calm  the  nerves. 

3) They  sure  do  take  long  walks. 

MALE  (CH) 

P/0  DAM  List  4A 

1  )They  sat  in  the  cool  park. 

2) Tom  left  home  in  disgust. 

3) We  watched  the  new  program. 

FEMALE  (KS) 

P/0  DAM  List  8A2 

1  )Sue  was  fast  on  her  feet. 

2) That  blue  copy  was  hers. 

3)  Floods  destroyed  your  attic. 

MALE 

ATC  Dialogue 

1) Westwind  111  hotel  november contact 

Atlanta  Center  now  on  126.77. 

2) 10  echo  golf  Washington  Center  Roger  29  and 
a  half  for  310  and  uh  are  you  direct  to 

Pulasky  or  uh  Knoxville. 

3) Eastern's  1 19  contact  Atlanta  126.77  Good 
Morning. 

4)  Delta’s  495  Roger. 

5) Piedmont's  962  contact  Washington  Center 
133.02  Good  Morning. 

FEMALE 

ATC  Dialogue 

1) Delta  95  heavy  contact  Atlanta  Center 

126.77. 

2) United  1021  Atlanta  Center  119(er).57. 

3)  Power  98  your  traffic  off  now  at  1 1  o'clock 
and  about  12  miles  northwest  bound  out  of 

330  climbing. 

4) lnterstate  2114  heavy  Washington  Center 
Roger. 

5) Lear  442  november  echo  traffic  12  O’clock 

5  miles  eastbound  of  330,  I’ll  have  lower  for 
you  momentarily.  I’m  going  to  need  a  good 
rate  of  descent  for  you. 

MALE  (RH) 

P/O  DAM  List  6A 

1) That  goose  layed  an  odd  egg. 

2) That  quiz  was  much  to  hard. 

3) Those  are  pudgy  old  men. 

A-3 


ECTIVE  L 


ROUP 


n 


SAMPLE 

# 


7 


8 


ISTENING  TEST  MATERIAL  FOR  PART  1 
SPEAKER  MATERIAL 


FEMALE  (MP)  1  )lnvest  your  money  now. 

P/0  DAM  List  8A  2)Take  all  the  chalk  with  you. 

3)This  man  was  knocked  out  cold.. 


MALE  (JE)  1  )She  saved  about  eight  cents. 

P/0  DAM  List  3A  2)His  clothes  have  some  false  cuffs. 

3)They  enjoy  loud  concerts. 


A-4 


PART  1.  Group/Sample  Codec  Sequence  Matrix 


i 

2 

SAMPLE 

3.  4 

6 

7 

a 

GROUP 

1  - 

A 

B 

C 

D 

E 

F 

G 

H 

2  - 

1 

J 

K 

L 

M 

N 

0 

c 

3  - 

M 

F 

H 

A 

L 

K 

J 

0 

4  - 

G 

E 

N 

B 

D 

1 

C 

M 

5  - 

B 

K 

0 

F 

A 

J 

L 

D 

6  - 

E 

G 

1 

H 

N 

C 

A 

F 

7  - 

K 

M 

L 

O 

1 

B 

D 

G 

8  - 

H 

N 

E 

J 

J 

O 

B 

K 

9  - 

N 

A 

F 

C 

G 

H 

1 

E 

10  - 

D 

L 

M 

N 

H 

A 

E 

L 

11  - 

C 

0 

B 

G 

K 

D 

M 

J 

12  - 

F 

1 

D 

K 

0 

G 

H 

B 

13  - 

J 

C 

A 

M 

F 

E 

N 

1 

14  - 

L 

D 

G 

E 

C 

M 

K 

N 

15  - 

0 

H 

J 

1 

B 

L 

F 

A 

PART  2.  Group/Sample  Codec  Sequence  Matrix 


i 

2 

SAMPLE 

1  4 

a 

a 

7 

a 

GROUP 

16  - 

1 

M 

D 

A 

J 

0 

C 

G 

17  - 

E 

F 

L 

B 

N 

K 

H 

H 

18  - 

A 

K 

B 

E 

c 

M 

N 

0 

19  - 

D 

J 

1 

L 

G 

F 

E 

A 

20  - 

0 

B 

K 

M 

D 

H 

L 

N 

21  - 

F 

C 

G 

J 

1 

B 

i 

E 

22  - 

N 

O 

C 

G 

A 

J 

D 

K 

23  - 

H 

L 

M 

F 

0 

A 

G 

B 

2^  - 

C 

E 

F 

N 

H 

D 

K 

L 

25  - 

M 

1 

J 

1 

L 

G 

J 

D 

26  - 

B 

H 

N 

0 

F 

E 

M 

C 

27  - 

K 

A 

A 

K 

M 

1 

0 

F 

28  - 

G 

D 

H 

C 

E 

N 

B 

J 

29  - 

L 

G 

0 

H 

B 

C 

F 

M 

30  - 

J 

N 

E 

D 

K 

L 

A 

1 

A-5 


SUBJECTIVE  LISTENING  EVALUATION  FORM 

FOR 

NEW  YORK  OCEANIC  AIR  ROUTE  TRAFFIC  CONTROL  CENTER 

AUGUST  4,  1988 


NAME: 

TIME: 


COMMENTS,  CRITICISMS,  &  SUGGESTIONS: 


A-6 


PART  1 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  1  -  4 


MG01  Sub  EvaJ  7/25/88 


A-7 


PART  1 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  5  -  8 


TEST 

SEQUENCE 

GROUP 

(SAMPLE) 

SPEAKER 

5  (1) 

FEMALE  1 

5  (2) 

MALE  1 

5  (3) 

FEMALE  2 

5  (4) 

ATC  MALE 

5  (5) 

ATC  FEMALE 

5  (6) 

MALE  2 

5  (7) 

FEMALE  3 

5  (8) 

MALE  3 

6  (1) 

FEMALE  1 

6  (2) 

MALE  1 

6  (3) 

FEMALE  2 

6  (4) 

ATC  MALE 

6  (5) 

ATC  FEMALE 

6  (6) 

MALE  2 

6  (7) 

FEMALE  3 

6  (8) 

MALE  3 

8  (5) 


8  (6) 


8  (7) 


8  (8) 


FEMALE  1 


MALE  1 


FEMALE  2 


ATC  MALE 


ATC  FEMALE 


MALE  2 


FEMALE  3 


MALE  3 


INTELLIGIBILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


ACCEPTABILITY 

5 

1 

EXC. 

GOOD 

FAIR 

POOR 

BAD 

PART  1 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  9  -  12 


TEST 

SEQUENCE 

GROUP 

(SAMPLE) 

SPEAKER 

9 

(1) 

FEMALE  1 

9 

(2) 

MALE  1 

9 

(3) 

FEMALE  2 

9 

(4) 

ATC  MALE 

9 

(5) 

ATC  FEMALE 

9 

(6) 

MALE  2 

9 

(7) 

FEMALE  3 

9 

(8) 

MALE  3 

10 

(1) 

FEMALE  1 

10 

(2) 

MALE  1 

10 

(3) 

FEMALE  2 

10 

(4) 

ATC  MALE 

10 

(5) 

ATC  FEMALE 

10 

(6) 

MALE  2 

10 

(7) 

FEMALE  3 

10 

(8) 

MALE  3 

11 

(1) 

FEMALE  1 

11 

(2) 

MALE  1 

11 

(3) 

FEMALE  2 

11 

(4) 

ATC  MALE 

11 

(5) 

ATC  FEMALE 

11 

(6) 

MALE  2 

11 

(7) 

FEMALE  3 

11 

MALE  3 

12 

(1) 

FEMALE  1 

12 

(2) 

MALE  1 

12 

(3) 

FEMALE  2 

12 

(4) 

ATC  MALE 

12 

(5) 

ATC  FEMALE 

12 

(6) 

MALE  2 

12 

(7) 

FEMALE  3 

12 

mm 

MALE  3 

INTELLIGIBILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


ACCEPTABILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


MG01  Sub  Eval  7/25/88 


A-9 


PART  1 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  13  -  15 


TEST 

SEQUENCE 

GROUP 

(SAMPLE) 

SPEAKER 

13 

(1) 

FEMALE  1 

13 

(2) 

MALE  1 

13 

(3) 

FEMALE  2 

13 

(4) 

ATC  MALE 

13 

(5) 

ATC  FEMALE 

13 

(6) 

MALE  2 

13 

(7) 

FEMALE  3 

13 

(8) 

MALE  3 

14 

(1) 

FEMALE  1 

14 

(2) 

MALE  1 

14 

(3) 

FEMALE  2 

14 

(4) 

ATC  MALE 

14 

(5) 

ATC  FEMALE 

14 

(6) 

MALE  2 

14 

(7) 

FEMALE  3 

14 

(8) 

MALE  3 

15 

(1) 

FEMALE  1 

15 

(2) 

MALE  1 

15 

(3) 

FEMALE  2 

15 

(4) 

ATC  MALE 

15 

(5) 

ATC  FEMALE 

15 

(6) 

MALE  2 

15 

(7) 

FEMALE  3 

MALE  3 


INTELLIGIBILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


ACCEPTABILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


PART  2 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  16  -  19 


TEST 

SEQUENCE 

GROUP 

(SAMPLE) 

SPEAKER 

16 

(1) 

FEMALE  1 

16 

(2) 

MALE  1 

16 

(3) 

FEMALE  2 

16 

(4) 

ATC  MALE 

16 

(S) 

ATC  FEMALE 

16 

(6) 

MALE  2 

16 

(7) 

FEMALE  3 

16 

(8) 

MALE  3 

17 

(1) 

FEMALE  1 

17 

(2) 

MALE  1 

17 

(3) 

FEMALE  2 

17 

(4) 

ATC  MALE 

17 

(5) 

ATC  FEMALE 

17 

(6) 

MALE  2 

17 

(7) 

FEMALE  3  , 

17 

(8) 

MALE  3 

18 

(1) 

FEMALE  1 

18 

(2) 

MALE  1 

18 

(3) 

FEMALE  2 

18 

(4) 

ATC  MALE 

18 

(5) 

ATC  FEMALE 

18 

(6) 

MALE  2 

18 

(7) 

FEMALE  3 

18 

(8) 

MALE  3 

19 

(1) 

FEMALE  1 

19 

(2) 

MALE  1 

19 

(3) 

FEMALE  2 

19 

(4) 

ATC  MALE 

19 

(5) 

ATC  FEMALE 

19 

(6) 

MALE  2 

19 

(7) 

FEMALE  3 

19 

(8) 

MALE  3 

INTELLIGIBILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


ACCEPTABILITY 


5  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


A-11 


PART  2 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  20  -  23 


MG01  Sub  Eval  7/25/88 


A-12 


PART  2 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  24  -  27 


INTELLIGIBILITY 


S  4  3  2  1 

EXC.  GOOD  FAIR  POOR  BAD 


TEST 

SEQUENCE 

GROUP 

(SAMPLE) 

SPEAKER 

24 

(1) 

FEMALE  1 

24 

(2) 

MALE  1 

24 

(3) 

FEMALE  2 

24 

(4) 

ATC  MALE 

24 

(S) 

ATC  FEMALE 

24 

(6) 

MALE  2 

24 

(7) 

FEMALE  3 

24 

(8) 

MALE  3 

25 

(1) 

FEMALE  1 

25 

(2) 

MALE  1 

25 

(3) 

FEMALE  2 

25 

(4) 

ATC  MALE 

25 

(5) 

ATC  FEMALE 

25 

(6) 

MALE  2 

25 

(7) 

FEMALE  3 

25 

(8) 

MALE  3 

26 

(1) 

FEMALE  1 

26 

(2) 

MALE  1 

26 

(3) 

FEMALE  2 

26 

(4) 

ATC  MALE 

26 

(5) 

ATC  FEMALE 

26 

(6) 

MALE  2 

26 

(7) 

FEMALE  3 

26 

m 

MALE  3 

27 

(1) 

FEMALE  1 

27 

(2) 

MALE  1 

27 

(3) 

FEMALE  2 

27 

(4) 

ATC  MALE 

27 

(5) 

ATC  FEMALE 

27 

(6) 

MALE  2 

27 

(7) 

FEMALE  3 

27 

(8) 

MALE  3 

ACCEPTABILITY 

5 

4 

3 

2 

1 

EXC. 

GOOD 

FAIR 

POOR 

BAD 

A-13 


PART  2 

SUBJECTIVE  LISTENING  EVALUATION  FORM 
FOR  GROUPS  #:  28  -  30 


TEST 

SEQUENCE 

GROUP 

(SAMPLE) 

SPEAKER 

28 

(1) 

FEMALE  1 

28 

(2) 

MALE  1 

28 

(3) 

FEMALE  2 

28 

(4) 

ATC  MALE 

28 

(5) 

ATC  FEMALE 

28 

(6) 

MALE  2 

28 

(7) 

FEMALE  3 

28 

mm 

MALE  3 

29 

(1) 

FEMALE  1 

29 

(2) 

MALE  1 

29 

(3) 

FEMALE  2 

29 

(4) 

ATC  MALE 

29 

(5) 

ATC  FEMALE 

29 

(6) 

MALE  2 

29 

(7) 

FEMALE  3 

29 

(8) 

MALE  3 

30 

(1) 

FEMALE  1 

30 

(2) 

MALE  1 

30 

(3) 

FEMALE  2 

30 

(4) 

ATC  MALE 

30 

(5) 

ATC  FEMALE 

30 

(6) 

MALE  2 

30 

(7) 

FEMALE  3 

30 

(8) 

MALE  3 

INTELLIGIBILITY 


4  3  2  1 

GOOD!  FAIR  POOR  BAD 


ACCEPTABILITY 


4 

3 

GOOD 

FAIR 

